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FOREWORD 

"HOW HAVE YbU USED NATIONAL ASSESSMENT MATERIALS?" ^ This 
question was asked of six educators ... Their stories 
unfold in the pages that follow. ' 

For readers not familiar with the National Assessment of 
Educational Progress (NAEP) , a brief overview may be helpful. 
The National Assessment is the first ongoing effort to obtain 
comprehensive and dependable data on education achievement on 
a national basis. The project reports on the current status 
of performance in various subject areas, including mathematics, 
rearding, writing, art, music, science and social studies, and 
also provides information about chahges in achievement over 
time. To help those interested in using NAEP findings, the 
project has developed reports and related assessment materials' 
(including assessment objectives and items used to measure 
achievement) that are geared to meet the needs and interests' 
of educators, legislators and the general public^. 

This collection of papers is "special" because the authors have 
"described, in their own words,- their first-hand experiences 
in using National Assessment materials as a resource to help 
solve a particular problem or concern. Their ^tories relate 
enthusiasm, inspiration, initiative and creativity. Reading 
between-the-lines, one can sense the ever-present challenges, 
frustrations and yes, even humor. 



We hope the stories from these six educators will show others 
some Of the many ways that NAEP findings and information can 
be used to improve the learner's education experience. Thank 
you, authors, for sharing your stories with us. 

0 
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THE HAWAIIAN^ CONNECTION: 
NAEB AND THE KAMEHAMEHA SCHOOLS 



James R. Brough, Ed.D. 
Director of Program Evaluation 
The Kamehameha Schools 
Kapalama Heights 
Honolulu, Hawaii^ 96817 



NAEP, an acronym that resounds with stirring vibrancy, is 
recognized by'informed educators in all 50states^as 
signifying the National Assessment of Educational ♦ Progress . 
Indeed, over the past 13 years, the National Assessment 

a 

movement has emanated, from its mid-America energy center 
throughout all''of the continental United States, It'also 
quietly ^ traversed ^the Blue Pacific, qrossed the sands of 
Waikiki and spread througriout the Hawaiian Islands. One of 
the strands of this migration climbed the lava hillsides of 
Kapalama Heights and entered^ the educational scene, of The 
Kamehameha Schools. In this untried and unfamiliar • 
'environment , the NAEP model has been methodically dissected, 
e^ach piece or component carefully scrutinize^l , and , 
ultimately reassembled . But, akin to the home mechanic who 
. boldly "decides to repair h?s owh car, we never put some of . 
the pieces back and o^ther parts ended up in new 'locations 
with different uses. Thus at The Kamehameha Schools, the 



National Assessment model and mate»"ials were metamorphosed 
] until the acronym NAEP might be also ^said to symbolize a New 

y Approach to Evaluating Performance. This paper will present 

just the highlights of this "Hawaiian Connection" --the 
coming together of The Kamehameha Schools and the National . 

Assessment of "Educational Progress.r? But first, a contextual 

■ / 

word or so about The Kamehameha Schools. 

• * '. • 

Kamehameha ' s origins are rooted deeply in Hawaiian 
history. The schools represent the living legacy of 
Princess Bernice Pauahi Bishop, the last descendent of. the 
royal Kamehameha .family. ^Mrs^ Bishop left her sizable 

c 

landholdings for the founding and maintaining- of these 
Schools. Classroom instruction, college scholarships, 
' summer camps-, Hawaiian culture, research ai?d developmen t'' 
the wide range of educational programs and righ traditions 
of the institution touch\the lives of residents throughout 
(^awai i . . 

■ J ■ 

Established to provide "education services for young 
peoDle of Hawaiian ancestry. The Kamehameha Schools have 
grown, dramatically since'their founding in 1887.' 'Today some 
2,750 students are enrolled full-time in a Kamehameha 
program, more than 9 , 60t) are" enrolled, part-time and another 
27,000 part icipate ^ in other programs of the scHools such as 
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demonstrations, lectures and workshops. In addition, 
findings of the Research and ^Development Division have the . 
potential to affect every chil'^ in the state— .and beyond. 

* a • . ... 

Such wide-ranging- activities make Kamehameha unique — 
one of .the most exciting education com.plexes in the country. 
Still, the schools, constantly seek new and more effective 
ways to fulfill the • education needs of. the young people .of 
Hawaii. The direction and scope of this continuing quest 
for quality is. fueled in large part by multiple kinds of 
evaluation and assessment information. However , un ique 
programs often require unique evaluation approaches -- 
approaches'and procedures that need to be "home grown." At ; 
\he same time, like-the hybrid pineapple or papaya, a local 
variety is inv igorajbed and strengthened by elements^ coming , 
from outside the family. National Assessment' has provided, 
this energizing strain. 

J 

But, where , why and' how has this "Hawaiian Connection" 
„. occurred? First, the "where." One .rather large-scale 
evaluation effort of the schools Campus Program is labelled 
the Exit Skills Testing Program (ESTP)i This project refers 
to an intensive in-house effort to identify expected student 
outcomes relative to School Goals For Stude nts and to assess, 
the degree to which the curriculum is • effectively and 
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efficiently 'assisting students irt achieving these ski 1 ]fs , 
The designation of "exit" skill's reflects the decision to ^ , 
base this curriculum evaluation on the tested performance .of 
graduating seniors. In oth^r words, ^ESTP focus,«s on theV 
ultimate product, of the schools anc} the» e)fe..tent to which 
these young men and^ women c|n demonstrate the expected 
levels of performance. The Exit Skills Testing Program is 
still in the infancy stage of development^ and to date, four 
secondary school departments English, Mathematics, 
'Science arfd Speech have developed and administered tests 
for assessment of their highest priority exit skills. 

The oth'er area where Kamehameha and NAEP have combined 
forces is in the development of basic skills assessment 
instruments for two alternative education programs for 
high-scj^pbl-age alienated youth in the public education 
system. These projects, one in Kona, Hawaii, and the other 

•V. 

on the "Friendly Isle" of Molokai, have produced' localized 
or "Hawaiianized" tests' i'n reading and mathematics to assess 
student acquisition of basic life -coping skills. 

The answer to the "why" question why did the "Hawaiian 
Connection" come to be reflects the unique nature of The 
Kameharr/eha Schools mentioned earlier. The Campus Program 
enrolls a selective population of relatively high-achieving 

'«« 

U 



students. The alternative schools serve largely youngsters 
with very limited academic skills. Both environments 
embrace a Hawaiian cultural milieu* Because of the 
specialized' nature of these populations, standardized and . 
other commerical assessment instruments h^ave been of limited 
use- for evaluating critical program oujbcomes. Thus, with a 
boldness tempered by realistic, trepid-a't ion , the schools 
accepted th^ unavoidable conclusion that locally developed 
evaluation materials and proced ures^ wer*^ required. .It was 
also instantly apparent that help would be needed — help 
1that was expert, resp)9nsive and econom.ical, but not 
prescriptive. In the search for this unl i.J<eiy|u cqmbination , 
Kamehameha , found NAEP. , , " 



Finally , -"how?" How has NAEP . prov ided this expert, 
responsive ^and e^conomical assistance? The analysis of the 
NAEP model that >rSs mentioned at the beginning of this paper 
led to- the use of four components as the cornerstones of the 
"Hawaiian Connection." These included: 



Materials / Tr?.1iiing 
Resources / Resources 



Rro'c^dures / Standards 




The first cornerstone used in developing new approaches 
for evaluating the exit^ skill levels of seniors and the 
performance in basic skills by alienated youth was securing 
and using NAEF materials directly. For example, The 
Kamehameha Schools obtained NAEP objectives booklets that 



were available in subject areas ranging from .art to English 
to social studies.* Teachers in the Campus Program reviewed 



articulate their ideas about the most important student 
skills. Words, phrases, and at times entire statements from 
the NAEP booklets were used in defining exit skills. 



Likewise, staff membeVs of the alternative schools used the 
•reading and mathematics objectives as guidelines for 



structuring basic skills tests in these two areas. The 
specificity of the objectives was« particularly appreciated 
by the alternative education staff, many of whom were "grass 
roots" individuals with highly developed vocational skills. 



cultural sensitivity and couni>eling prowess, but with 



minimal formal educationf. 

The use of "released" test items,,was a second way in 
which NAEP materials were directly applied in the Kamehameha 



evaluation system. The schools obtained all NAEP released 
exercises (items) and these served as a "raw" item pool in 
many subject are^s. Following the difficult and tedious 



the statements in the booklets as they struggled to 
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task of matching items to instructional objectives, teachers 
selected a number of NAEP items for inclusion in tests. 
Several exam'ples follow: 

Percent of NAEP 



Test Area ^ Items Used 

■ High School Math Exit Skills 80% 

High School Science Exit Skills 12? 

Alternative School Basic Skills [ 

Reading 26? 

Mathematics ^2? 



In the above four tests alone, 116 NAEP released exercises 
were used. Only the naive or masochistic would fail to 
recognize the enormity of this contribution compared to the 
time, frustration and professional trade-offs that accompany 
the writing of 116 original test items by local classroom 
teachers who are already fully employed. 

A third relatively minor use of NAEP materials, but one 
of major convenience, was the loan of "full size" graphics 
by NAEP for the items included in 'the exit skills test's. 
This permitted the diT^ect dupl ication of the exercises and 
eliminated the trouble and expense of enlarging the ^ 
published exercises prior to printing the test booklets. 



In addition to the direct application of materials, a 
second cornerstone of the "Hawaiian Connection" was the use 
of NAEP as a training resource. For example, the objectives 
and released exercises were used as models in a number of 
disciplines, even though no specific objective's or items 
were replicated. This application was used by the teachers 
developing an exit skill writing sample. They reviewed the 
NAEP writing sample stimul i but opted to create their own 
actual stimuli, though based upon the NAEP model. .Another 
case in which the e^tercises and objectives were used as 
modelS'^rather than in direct application was in the 
development of ah exit skills test in Hawa i ian cul ture . 
This area, of course, demanded local development, and the 
existence of item and objective formats was somewhat 
comforting to the Hawaiian Studies staff who' were literally 
entering into an unknown world. 

A National Assessment training resource that was used 
with undisputed positive impact was that of interpersonal 
contact. Even though Hawaii is thousands of miles away from 
Denver, a number of Kamehameha administrators and evaluators 
visited with key personnel from NAEP. In conjunction with 
attending conventions, extend ing personal vacations or 
specifically scheduling conferences, the "Hawaiian 
Connection" has maintained its vitality ov^r the years. In 

8 



, ^ ...^ 

this regard, the annual .Large Scale Assessment Conference 
has had representation from Kamehameha for the past five or 
six years and is viewed as a consistently superior 
ex per ience . ,j .= 

Adopting several pieces of the NAEP procedural model 
became the third cornerstone of the NAEP - Kamehameha 
relationship. For example, the Exit Skills Testing Program ; 
utilized -the concept of identifying differential difficulty 
levels of items based upon dge level. Altho'ugh the exit 
skills tests are, by definition, for assessing graduating 
seniors, eighth- grade students were also tested with the^ 
same items and achievement results compared for the grade 
-levels. This procedure closely parallels the 13-year-old 
and 17-year-old testing program of NAEP. Several other 
National- Assessment procedures were used in the Exit Skills 
Testing Program including setting the testing groups at 15 
students and employing NAEP scoring methods in scoring 
writing samples and essay a^nd oral responses in various 
subject areas. ^ ' . ' 

The final cornerstone to be mentioned is the element that 
provides some degree of confidence and securj^ty to staff_aa 
they forage in the fields of local test development. 
Whether in the exit skill are-na or assessing per formance in 

6 
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basic skills, local staff members need some validation ,that 
what is produced is of quality. A standard is needed. The 
National Assessment- model has provided such a standard in at 
least two din/ensions. First, the body of NAEP objectives' in 
a given subject area was used as a standard of 
comprehensiveness for the sets of exit skills that were 
drafted by departmental areas-. Second , the actual student 
results on both the exit skills tests and the alternative 
education 'basic skills tests would have lacked any reference 
points .had it hot been for the NAEP it^-ms that were 
included. The national "norms" for ages 9, 13 and 17 on 
NAEP items provided one valuable perspective. 

So, the "Hawaiian Connection" has been made. The NAEP 

"J 

cornerstones have provided a structure for a new approach- to 
evauiating performance at The Kamehameha Schools.. The 
"connection-," however, is .a very young one — one that 
epitomizes the term "pilot." Yet, the staff of the schools 
is growing in confidence that the Exit Skills Testing 
Program will indeed provide guidance in curriculum 
modification and development. An equal optimism is building 
within the alternative education programs that localized 
assessment of basic life skills will indeed lead to. more 
effective programs and more successful students.* Fgr the 
contribution made by the National Assessment of Educational 
Progress: 

- 10 
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USING NATIONAL ASSESSMENT IN DOCTORAL RESEARCH 

Stanley E, Easton-, Ed.D. 
Associate Professor of Secondary Education 
Montana State University* 
.Bozeman, Montana 59717 

o 

- o . 

Over the past several years a few doctoral candidates at 
the University of Mississippi and I developed and. pursued an 
interest in describing some of the educational achievements 
of rural, youth. Specifically, we investigated the 
c i tTze n^M prTn7~so cTa l"~sTud i e inments of~13-year-oids 

attending schools in some rural areas of Mississippi and 
South Carolina. We use'd test items released to the public 
domain by the National Assessment of Educational Progress 
(NAEP) to survey the knowledge, skills and attitudes of 
selected populations. "National Assessment sampling and 
data-gathering procedures were studied and adapted to these 
local surveys. 

9 

The purpose of this article is -to share highlights of 
these experiences in. dissertation research with interested 
readers who are already familiar with National Assessment's 
mission and nature. 



•Formerly at the University of Mississippi 

13 ' 

In 



Backgro'Dnd 

The idea of using National Assessment in doc toral stud ies 
occurred to me while listening to d prjesentation^ by Bob L. 
Taylor at the 1973 annual conference of the Association for 
Supervision and Curriculum Development. Taylor, professor 
of education at the University of ^Colorado , tlescribed the 
NAEP model and discussed its application to state and local 
assessments. Also, he talked about the implications of 
using the mo'del for curriculum development anc^ 
a c co^'unXTBTlT^^^ l"^w¥s ^TnTr Fg^Jed^^^ b se r v a t i olT 

that National Assessment ^would be most useful in , situations 
where the focus is on program evaluation, rather than on 
judging the per formance of individual students or teachers. 
As the accountability rfibvement gained momentum, I thought it 
would be worthwhile to demonstrate the efficacy of an 
alternative to the norm-referenced standardized achievement 
-testing of thousands of students.^ 

In my work with doctoral candidates in the field of 
secondary education, I was concerned that they identify 
research problems for which they could obtain data readily 
and which, ^t the same time, were of interest to a wrder 
community of scholars. Surveying the progress of local . 
groups of students in relation to the goals and objectives 
developed anu validated by National Assessment seemed to 
satisfy my emerging criteria. 



Therefore, I began to introduce the idea to prospective 
dissertation writers' one^ a't a time over a period of years. 
All of the candidates had interests in citizenship and 
soci-al studies education, and were acquainted with rural 
schools in the South. Some of them chose to research other 
kinds of problems, some of them with other professors. „ To 
date, three doctoral ' students have elected to survey .rural 
13-year-olds in either citizenship or social studies. Two 
of the* .studies were conducted in northern Mississippi and 

t ~so u CTT" t: a r^^ er t a k i n g t h e s e - ^ 

investigations, we felt we would be making small 
contributions to knowledge in an area — rural education — 
that has had an abundance of information concerning the 
■ environment for teaching and learning and a scarcity of 

facts about the outcomes cf schooling. We hop°ed in addition 
that candidates might develop some marketable research 
skills. ■ ' " 

A Summary of Procedures Used in Three Dissertations 
Approximately 12 to 1 5 months were required to see each 
study from conception through final acceptance- by a 
dissertation committee. Each paper represented a major 
^^\investment of time, effort and other resources on the part 
oKa single researcher, working with only minimal financial 
assist^ce. A recapitulation of the procedures used in the 
three studies is presented in th.e following paragraphs. 



These steps suggest those that mighti be followed by other 
researchers wishing to apply NAEP in new situations. 

The Research Proposa^l 

An important early step was the dev.elopment of a formal 
research proposal . The proposal included a concise 
statement of the problem to be investigated and listed, 
specific questions to be answered by the study. The 
researcher identi f ied the assessment area (citizenship or 
social studies) to be addressed and the target population 
(example: 13-year-olds attending schools in communities of 
less than 3,500 persons within 7S miles of Oxford, 
Mississippi) to be surveyed. A 3.ampl ing design was 

presented in the proposal, along with an outline of 

■ 1 

procedures to be followed in collecting and analyzing data. 

Assessment Material s 

The materials used to conduct these a"ssessmen'ts were 
drawn from the items National Assessment released to the 
public following . publication of the results of the first 
nationwide surveys of citizenship and social studies. NAEP 
provided copies of the released exercises ( items) and 
directions for making a narration tape to accompany the 
group-administered exercises. We obtained recordings of 
musica] selections and copies of pho'tographs that had been 
included' in the stimulus materials and borrowed ^ pript of a 
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filmed episode NAEP had used to elicit responses reflecting 
attitudes. . 

Exercise booklets were design>c>d and prepared so that no 
student would be given all of the exercises or spend more 
than a single class period on the assessment. Some of the 
clusters of exercises were to be administered to groups of 
.students, others to individuals. The difficulty levels of 
the booklets were equalized in accordance with NAEP 
p r axitic e ... Je f er ei^ -to , Na ti o n a 1 As s e s sm e n t_£ ub 1 ica tlo n s a n d 
a few long-distance telephone consultations with NAEP's 
Department of User Services were helpful in this 
developmental" stage of our studies. 

The Dissertation Committee ^ 

A "necessary evil" peculiar to doctoral research is the 
dissertation committee. Given that, it is essential- to the 
sanity and general well-being of«^'all concerned to form a 
supportive group of graduate faculty members to guide the 
candidate's research effort . For our studies, professors 
with interests in citizenship or social studies, and 
assessment and research design were recruited. Prospective 
committjse members were^ consul ted as each proposal was being 
developed. When an assessment plan was nearly complete, a 
dissertation committee was organized and constituted ^\ 
formally. 



, The committee approved the proposal with amendments. 
This officially sanctioned document thereafter served as 
both blueprint for the study and contract between the * 
candidate and the comiTiittee . r Each researcher continued to 
consult committee members for the duration of the project. 

Conducting the Assessment 

To carry out the planned assessment, the doctoral student 
first identified the schools in which the target population 
was enrolled. Schools were selected randomly for the 
a^¥e¥snrenT^san^ Administrators of the , sample schools 

were contacted, and lists of all 13-year-old students were 
obtained.. Letters typed on University letterhead stationery 
we're sent to the administrators of ' the schools to solicit 
their participation. Appointments for the administration of 
the assessment exercises were made with the principal of 
each sample school. Students to be included in the survey 
were selected randomly by the researcher from the lists of 
1 B-year-olds . 

On the appointed date and time, the researcher arrived at 
the school with the materials and equipment necessary to 
conduct the^ assessment . The principal either directed or 
escorted the doctoral candidate to the area set aside for 
the assessment and sent for the selected students. 
Following a brief explanation of the study, the exercise 
booklets were administere'd to the students. 

18 
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Re porting Assessment Results 

Responses to the assessment exercises were scored 
according to;NAEP criteria. for acceptable responses. The 
percent of students giving acceptable answers was tabulated 
for each exercise. The percentages of males, females, 
blacks and nonblacks giving acceptable responses were 
tabulated for purposes of comparison. The- performance of 
each local sample was compared with that of the NAEP extreme 
rural sample* Results and compar isons ..were presented for 
—e^Sh^o-f^t h-e^a s.s.e.s sm,en t . . ex erc.ise s ad m i n is te r e d 

(•36 citizenship exercises or 58 social studies exercises). 

Producing an assessment report in dissertation form 
proved to be especially laborious, given the 
exercise-by-exercise approach we had elected to follow. The 
doctoral candidates found themselves compelled to write in 
an unfamiliar style. they had to develop their own patterns 
of narration since they were pioneers in the writing of 
assessment reports for dissertation committees. Advising 
• the researchers was ^difficult because the projects were 
' .novel to committee members. Perhaps there are better, less 
cumbersome approaches to writing such stud'ies. If so, they 
should be eagerly sought and warmly embraced'. 

Outcomes From Using NAEP in Doctoral Studies 
In g'eneral, the rural 1 S-year-.olds surveyed by the 
doctoral students perlTormed less well than the earlier NAEP 
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extreme rural samples on citizenship and social studies 

exercises. Such findings were not una^n t ic ipated since the 

* ' •. . ^ 

samples we r is'' taken in the southea^stern. United States, the 

region that consistently has scored lowest 'on all subject 

areas assessed by National Assessment. However, one might 

expect there to be exceptions to the rule, and local 

assessments are the onl.y way to determine the standing of 

local school curricula and students with resipect to National 

Assessment's goals and objectives. 

These studies demonstrated the feasibility oT applying 
National Assessment materials and techniques in doctoral f 
research on local schools. The procedures followed produced 
considerable ' amounts of new information about ed u6at ion 
programs in selected rural areas with economy and , 
•efficiency. In one of the assessments, administration time 

was reduced by 86 percent of what would have been required 

■ ^ 

to test the entire .population with a norm-referenced 

. - " , * i> * 

standardized test of 70 minutes duration. 

The three doctoral candidates contributed some 
information concerning the outcomes of rural schooling to 
the rural education literature.^ In the process they gained 
first-hand experience with NAEP exercises and methods, a • 
working knowledge of National Assessment on a pedUGed scale,^ 
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By personally gathering the data for'^heir research, they 
saw rural students and their schools and developed a "feel" 
for the people whose performance they were document ing . and 
reporting. ^ 

There appears to be a potential benefit to students in 
local schools. .by assessing their performance on NAEP 
exercises. The vision .of university scholars working 
hand-in-hand with school of ficiars to determine the extent 
to which youngsters are meeting widely ' recognized goals is a 
pleasant one. It is, however, one which did not materialize 
in the cQurse of these studies. More deliber'kte efforts to 
involve local school administrators in using the survey 
results and to obtain ^ follow-up information on curriculum 
improvements would be appropriate in future applications of 
Natibnal Assessment to dissertation research. 
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STRATEGIES-FOB INCORPORATING NAEP MATERIALS 

IN "RESEARCH TRAINING PROGRAMS^ 

IN HIGHER EDUCATION 

•Robert K. James, Ph.D. 
Prbfes'sor of Science and Env ir'onmen tal Education 
Department of Curriculum and Instruction 
Kansas State University 
Manhattan , Kansas 66506 

Graduate students developing research skills in today's 
world require a varie,ty of research-related competencies. 
The need for statistical and research design capabilities is 
obvious and is almost always provided for in formal courses. 
Students also, need a broad" knowledge of the literature 
related to their area(s) of specialization. In « par t icul ar 
they need to be familiar with "bench mark" studies that can 
provide foundations on which .to. build their own research ^ ^ 
programs. In many instances they will need to be skillful, 
in developing test instruments and/or in locating resources 
for instrument development. Finally, students must develop 
theability to interpret and apply research findings. It 
seems likely that all but the first of the competent- ies 
listed above often will be gained in informal, one-on-one 
settings under the leadership of a major profes.sor.' 
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students doing graduate work in the area of science 
education are particularly fortunate in regard to'the 
literature resources available. There have been a number of 
broad-based studies in science education that have provided 
a picture of the current status of science education 
nationally. Weiss (1978) and Stake and Easley (1978) have 
completed studies of the status* of science teaching, 
curriculum alnd current practice.' Others (Helgeson, Blosser 
and Howe, 1977) have., provided a comprehensive summar'y of the 
literature since 1955. , Several groups have attempted to 
digest and react. to these results. Project Synthesis (Harms 
and Yager, 1981), has made a valuable attempt to provide the 
next step by suggesting the directions in which science 
education should move. 

; A3 valuable as these studies are, none of them has 
supplied the broad long-term assessment of science learning 
and science-related studies that is provided by the National 
Assessment of Educational Progress (NAEP). The National 
Assessment studies, conducted since 1969i -give a sampling of 
the science knowledge and attitude.s of students at selected 
age and grade levels. In addition, careful attention to 
instrumehtation and sampling procedures in repeated 
assessments makes it possible to observe trends in the data 



ifor 



a learning area. Th'es^e 3tud les , .com.b,iri.ed wi th the 
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others listed above, provide the avenue through which we can 
begin to furnisb most of these competencies presented at the 
beginning of the article. . 

In using NAEP materials with my students, I point out 
that the NAEP studies provide an indication of the knowledge 
levels of selected age groups and that repetitions of these 
studies have provided a measure, on a nationwide basis, of 
the trends in science learnings and attitudes. Further, I 
point out that data are available for regional populations 
and that some states have used all or parts of. the NAEP 
instruments to determine local and statewide achievement 
levels. I might discuss the .possibil ity of using NAEP items 
in assessing the knowledge or attitudes of students in our 
own researchv ef forts . 

NAEP studies also provide science educators with data In 
the auxiliary areas of ' env ironmental and energy ^ed ucat ion . 
Further, knowledge of the data on support skills such as 
reading and mathematics is valuable to science educators as 
they conduct their work. It is imperative that students 
stay abreast of NAEP activities. Students discover that the 
NAEP Newsletter ^is a. valuable and convenient sourpe of 

inf-ormation abbut NAEP^s work and contiains summaries of 

' * ' * ... J, 

assessment findings;; *They are- urged to add\.the ir. names to 

. ■ • \ ■ • I 

\ . 
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the bailing list (free) so that they will be able to follow 
NAEP's activities. The Newsletter forms a primary avenue to 
my students' overview of NAEP studies. 

Typically, I grab my folder of Newsletters and point out« 
to students items such as the following : 

(1) Fall 1980 issue contains a 1969-70 through 1980-81 
timetable of the tests. 

(2) April 1980 issue reports on women's mathematics 
achievement . 

(3) The Winter 1980-81 issue says that science will be 
added to the 1981-82 assessment. 

(U) The December 1978 issue reports a summary of the 
assessment of the energy-related knowledge of young adults 
aged 26-35 and shows that adults lack facts in this area. 

(5) Various issues announce the variety of products NAEP* 
provid'es, including summaries of results, objectives for 
each learning area, the released assessment items and a 
variety of kinds of analyses such as comparisons for 
public and private schools. 

(6) The Oqtober 1978 issue reports a summary of the NAEP 
health knowledge stud y , 

and so on.. • . , i 

. c 
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A particular situation in which we applied NAEP knowledge 
was in regard to a Department of Energy Faculty Development 
Grant. Thirty science and social studies teachers were to 
be brought to the Kansas State University campus for an 
intensive two-week energy education program. At the time 
the grant proposal was 'prepared , 'we decided to use the 
"Energy Inventory" prepared by the Biological Sciences. 
Curriculum Study. It became apparent that this instrument, 
was not going to assess the full range, of cognitive and 
attitudinal learnings we hoped would be the outcomes of the 
training program. Further, we wantedan instrument that had 
been designed for use with adults. 

We ordered. two NAEP publications: Technical Information ' 
and Data from the 1977 Assessment of Health, Energy and 
Reading ; and Energy Knowledge and Attitudes ; ^ A National 
Assessment of Energy Awareness Among Adults .- These reports 
were examined, and we agreed that an instrument^made up of 
the NAEP item.s used in this study was appropriate for both 
the content and population of our ins^titute. • 

An instrument was developed and administered. While the 
results have Hot , been fully analyzed, simple comparisons 
show significant, gains on a pre- and post-training 
'administration." Th^ use of NAEP. items and materi'a^Xs clearly 

■•' . 
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demonstrates to our students the value we place on NAEP 
efforts and shows them how NAEP materials. can be employed 
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THE CHEYENNE MOUNTAIN EXPERIENCE 

Thomas B. McCord, Ed.D. 
Director of Personnel 
Air Academy School District #20* 
806.1 Horizon Drive 
Colorado Springs, Colorado 80918 



• It is my intent to discuss briefly how local school 
districts may work, cooperatively with the National 
Assessment of Educational Progress (NAEP). I^ shall do so by 
describing a project which was undertaken ^ and completed that 
benefited our . particular school district. You shoUld 
understand one thing here at the outset I am not a ^ * 

philosopher, never have been, and have no plans to be! I 
will not try to deceive you by telling you that here is a 
foolproof method by which you can solve all of your 
educational probl^ems. I am\in the trenches, like most of 
you, with the usual amount of pressure and criticism, so my 
approach is practical. I 'should mention that as a result of 
our cooperative project with NAEP, we plan to use their data 
on a continuing basis in the future. I suspect that many of 
you would be able to\s6 their data effectively in your own 
programs . 



\ 



•Formerly Assistant Superinte,ndent fof Instruction', .Cheyenne 
.Mountain ..Schools ' \ 
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Qheyenne Mountain School District 12 is a small, suburban 
district on the southwest edge-of Colorado Springs. We 
enjoy a relatively high per-pupil expenditure, a low 
pupil-teacher ratio and students who come from 
college-oriented homes. Since approximately 85 percent of, 
bur graduates go on to college, we definitely are coU'ege 
preparatory. As in most districts, problems exist. The 
problem that brought about our relationship with NAEP was: 
our students tratl i tional 1 y score two to four grade levels 
higher than national n^rms on standardized tests. Maybe we 
should not compl^ain^ On the other hand, it occurred to some 
of. us that'our stud ents should score well on these tests, 
particularly when the total environment of our students is 
considered. The question arose: how do our students stack' 
up with students from around the country whose socioeconomic 
circumstances are similar? 

I had known of National Assessment since its.early days, 
but like so many others, I was uncertain as to whether it 
was operating in the best interests of public education. I 
remember, as a matter of fact, that I was apprehensive about 
the project^s intentions. At thispoint, I am very 
comfortable with the idea that. NAEP does have a -wealth of 
experience, expertise and data that can be useful to school 
administrators.- Af ter d is^covering that NAEP headquarters, 
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were in Denver, I wandered into their offices and expressed 

%? ■ ^ ■ ' 

..nfy concerns. 

NAEF staff and I put together a plan whereby Cheyenne 
Mountain '9-year-old students would be assessed in 
mathematics, and the results then compared with NAER's 
advantaged-tirban group. Our^ students fit nicely into this 
particular category. We found that NAEP could provide 
specific information for comparisons: i.e., data by 
geographical region, sex, size and type of community, level 
of parental education and so forth. That was just what we 
had been looking for, an available data bank that gave us 
the capability of making our desired comparisons . 

Financing is difficult these days as you know; therefore, 
we had to devise^ ways to keep expenses DOWN* Obviously, the 
best way to do that &as ,to try a do-it-yoijrself system. We 
decided to prepare our o.wn exercise booklets. (We called 
this startling new process the MIMEOGRAPH TECHNIQUE.) The 
rrrost unusual aspect of^the project was bur use of volunteer 
parents to score the booklets and tabulate results. The 
plan was essentially set. Now, for the administration of 
the assessment . 
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Since we are a small school district, our total 
9-year-old student population was only 120 youngsters. 
Following very closely the procedures used by NAEP in its 
nationwide assessments, we administered 65 items that we had 
selected for inclusion in the item booklet. These wer'e 
'exercises representing each of NAEP's five major mathematics 
goals. It is important that school district learner 
objectives b^, compatible with NAEP'^p. Ours fit beautifully. 
We were ex t rem el y careful in cond ucting the entire project 
in order to keep our statistics clean. We wanted no room 
for criticism . ^ NAEP procedure called for a recorded voice 
to present all instructions and read each exercise in turn 
to the students, even thoDgh all directions were printed in 
the^ booklet. NAEP insists on minimizing the possibility of 
student misunderstanding, so we made the tape following all 
instructions and , timing information established by NAEP. 
The assessment was structured so that a specified amount of 
response time was allowed after each exercise had been 
orally presented. Total assessment, time was about 55 
minutes.'. . . ^ 

.f * . > . 

What did we do after, test administration? If you will 

recall, I mentioned that' volunteer parents were a very 

important part of the project. Since the building 

principals knew those^^rents who were willing, capa.bie and 
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interested in participating in such a project, the selection 

was left to them. Twelve parents representing our thr^e, 

elementary schools volunteered their time to score the 
assessment booklets. 

We decided that a training session for the scorers was . 
critical. We had no idea of the time that would be required 
to prepare the scorers for their task. Neither were we able 
to estimate the number'of hours or days that would be 
necessary for iscoring and' tabulation. Jn our training 
session, the r e ason_jJjil^ the project was explained, a 

brief introductory filmstrip depicting NAEP history and 
capabilities was shown and the scorers were given 
step-by-step instructions on exactly how to score the test 
exercises. In a very shor,t period of time tjie scorers 
understood tiieir role in the project and Were able to score 
in a consistent and reliable manner. 

We ail were surprised when the scoring process was 
compl^eteV in one day. Twelve volunteers scored over 7,000 
- exercis^/S — that is over 600 each!. Scoring was completed 
in an ordinary working day, and it was done at very little 
cost to7 the district other than coffee and doughnuts! We 
were 'C:c(mpletely satisfied with the accuracy of the scoring-; 
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Following the scoring session, we designed ' forms on which 
we could tabulate results. About a week after ^ the scoring 
s e s sion, the same~1 2~vo lunteers returned toTialiT-. tabulate 
scores on our homemade form?. As mentioned previously, we 
had no way of accurately estimating the hours that would be 
utilized "^in the tabulating process. But, again, we were 
pleasantly surprised. Tabulating took approximately the 
same amount of time as was required for scoring. By this 
time our very dedicated volunteer parents had completed the 
nasty part of the project in much less time than any of us 
would have guessed, saved the district a significant amount 
of money and had become extremely interested in the final 
results. , . 

4 ■ - I 

/ 

■ • , / 

The entire process was considered a pilot project to 
determine whether it would produce worthwhi results . 
Since it was a pilot project which had some possible 
implications .for NAEP, National Assessment assumed the 
expense of data entry and computer time./ Please do not- 
expect that luxury in the future. Our .tabulation sheets 
were sent to NAEP and, In a relatively/ short time, printouts 
were, in our hands. / 

t 

Two very important questions were answered: (1) Cheyenne 

Mountain 9-year-old students Compared favorably with their 

/ 

* ■ ■ *• 

■ / 

. 36 



n¥tXon vdrd¥ coW^^ (2 ) Na t ional 

Assessment did have the capability of wording closely with a 
small local school district"" in a relatively inexpensive* 
manner. Beyond these two answers another important aspect 
of the project came to light: a number of doors or avenues 
were opened for future consideration. Let me give some 
examples of ways in which districts may use NAEP data: 

- First and basic to the entire idea, students may be 
compared with' others like themselves on each of the items*. 

- Data are available for comparisons by region of the 
^country, size and type of community, sex, grade level, 

level of parental education and o so forth. 

- Information may be obtained from each building for 
comparative purposes. In our case, each principal had 
prior knowledge '^of and supported this approach. 

- It is possible to obtain information concerning the 
students, in each classroom; however, in our case we chose 

to postpone that particular avenue until spme time in. the 

ft 

future. 

- If exercises are consistent and compatible with 
distfictwide^ learner objectives, it is a simple matter to 
pick out program weaknesses, strengths, trends and so 
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forth, especially as they relate to specific exercises* or 
groups oV related exercises. 

- Curriculum change or alteration can be made auickly and 

\ 

so 

effectively after an assessment project such as this. 

- You can '^economize! We estimated that our project COULD 
have cost a minimum of $5,000 — it cost us less than 
$250! ^ . • 

-"^The FLEXIBILITY is, in my estimation, the single most 
important feature of our project. You may design your 
assessment to fit your own special need s , collect data 
that are important to you and then decide how you wish to 
use results in improving, youl" programs. 

What has happened to us as a result of our cooperative 
project with National Assessment? 

\ 

- We have assessed both 13-year-old and 17-year-old students 
in mathematics. 

- We have conducted a science assessment of our 13-ye3r-oid 
students. 

- We recently completed a writing assessment of 
approximately 150 13-year-old junior high students. 

- Our intentions are to continue on a cycle basis and 
reassess later. We also plan to cond^jct initial 
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assessments in other areas, such as reading, social 
studies, music, art and career and occupational 
development. 

- We deliberately involved teachers on a non threaten ing 
basis; therefore, all teachers cooperated and most feel 
good about future implications for improvement of our 
instructional programs. 

- We will pontinue to involve as many people as possible and 
be completely open in reporting results. Our School 

(I 

Improvement Committee (Accountability), faculty, 
administration and^ Board of Education are advised 
periodically of new assessment projects and progress made 
on old projects. As a matter of fact, these groups are 
actively involved aa much as possible. 

- Finally, there is s6me excitement over the fact that we ' 
now do our own computer work. Our high school mathematics 
department chairman developed a pomputer program that 
provides the information that NAEP initially provided, and 
this will be an ongoing project for the computer 

> programming studentis in the high school. 

We believe that NAEP has something to offer school 
districts if staff and administration have .the desire ^to 
learn mo^e about their students. I am not saying that 
NAEP's approach should replace standardized testing. 

i 
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Rather, I believe that the NAEP approach could be used in 
conjunction with standardized testing programs, thereby 
giving district personnel an opportunity to look at things 
in another light. It cfn be a valuable tool in 
strengthening curricular programs and a very definite help 
for teachers in goal- or objective-oriented situations. 

I have given you no magic, and very, little philosophy, 
but I hope that the simple project which has been described 
may be of help to you in your future plans. Neither have I 
given you any .sort of formula which is suited to each of 
your situations. I do believe, however, that ^there is such 
flexibility^**S^ilable in working with NAEP that you are free 
'to piak and choose and, consequently, devise a plan of 
action to fit your own unique needs . 

School districts CAN accomplish many things without a 
great deal of money, highly paid consultants or elaborate 
research facilities. People in Cheyenne Mo un tain School 
District have proven this. 
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PROGRAM DEVELOPMENT UTILIZING NATIONAL ASSESSMENT MATERIALS 
IN THE MONTEREY PENINS.ULA UNIFIED SCHOOL DISTRICT 

Lloyd Swanson 
Director of Evaluation 
Monterey Peninsula Unified School District 
700 Pacific, P. 0. Box I03I 
' Monterey, California 939M0 

e ■ 

In the late 1960s, a survey of the parents in the 
Monterey Peninsula (California) Unified School District was 
conducted to gather information on their perceptions of the 
district's education program. The results were not unlike 
the more recent Gallup Poll findings — the basic skills 
programs needed improvement along with a need for better 
discipline in the classrooms and playgrounds. An adjunct 
part of this study was a survey of the local business and 

agency employers. Their messages were clear — too many of 

\ 

1 

the high school graduates could not adequately read, 
compute, spell, write or follow directions. 

This information set the stage for convening an array of 
curriculum review committees in the early 1970s to prepare 
recommendations for the resolution of the concerns 
identified. The ensuing dialogue was open, honest and 
filled with emotion. Some factions of the English teachers 
wanted the students' only to be creative in their written 
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compositions. The skills of writing with complete sentences 
and coherent paragraphs were of secondary and tertiary 
value. Others held a different view. Similarly, there were 
factions among the math teachers. Some held that computings 
was of primary importance with little or no time to be given 
to the applications of math in problem situations; others 
felt problem-solving skills were of great importance. 

As these deliberations continued, the essence of what the 
parents and employers were saying tended to be lost or 
interpreted to fit particular biases. The need to look to 
other sources as ameans of. resolving the issues led to 
National Assessment. 

Inasmuch as the major impasse was centered in writing, 
thfe NAEP reports on the 1969-70 writing assessment were 
obtained. The description of the methodology employed in 
preparing the objectives, the test items used, the national 
scope of the assessment and the findings impressed those 
involved in the deliberations. Unquestionably, this review 
of the first writing assessment report did more to reduce 
some of the well-entrenched provincialism among the staff 
than two years of highly charged committee meetings. A kind 
of consensus finally emerged, and as an unexpected 
consequence a proposal was made to model the NAEP assessment 
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approach in the district. No longer were community concerns 
to be essentially ignored, or formal assessments of student 

1 ^ ; : 

learning perceived.^ to be necessary threats or infringements 

'i . ■ ' . ' 

on the instructional, methodology . 

The development of ^.cr iter ion-referenced objectives in 
reading, writing and rnathemat ics was to be followed by 
assessments of students* progress to be conducted at the end 
of grades 3, 6, 9 and 12. Subsequently, the Board of 
Education adopted a policy requiring the attainment of 
specified minimum proficiencies in reading, writing and 
mathematics as a requirement for high school graduation, 
beginning with the class of: 1972. This policy predated the 

current proficiency movement by a number of years. 

\. - ■ - 

The basis for the district's writing objectives and test 
items was found in the NAEP released materials, from the 
1969-70 assessment. Since the reading and math assessments 
had not yet been conducted, district-developed objectives 
and test items were used in these areas. Later, as 
materials from the NAEP 1970-71 reading and 1972-73 
mathematic assessments became . available , modifications" 
and/or additions were made to the district's objective and 
item banks . 
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There is no doubt that this early utilization of NAEP's 
approach to education assessment had a significant impact on 
the instructional program development and implementation an 
our district in the following ways: (1) the beginning of a 
better balance between program processes and program 
outcomes w^s established and (2) modifications were made in 
teacher staffing arrangements to be^tter ensure the 
attainment of the desired learning outcomes. This is to say 
that the interjection of the NAEP format into the district's 
instructional program deliberations stimulated both 
philosophical and empirically based thinking about "how do ■ 
we know" knd "what does it mean" in curriculum' program 
planning , implementation and evaluation. 

l^e second application in utilizing the NAEP material 
began several years ago with the 1975-76 citizenship/social 
studies objectives and released test items. This 
application took the form of assessing all graduating 
seniors with the available NAEP test items along with a 
number of locally developed questions. A matrix sampling . 
format was used in which the items were. divided into three 
sets of tests. The results were reviewed by a committee of 
junior and^senior high school social studies teachers. 
These data in turn provided a basis for a year-long review 
of the district's social studies curriculum at the junior 
and senior high school levels. 



The payoff from this exercise was in the form of (1) an 
increased cognizance among the teachers of the differences 
between the district ' s written or prescribed social studies 
curriculum and the descriptive curriculum, or what is being 
taught in the classroom; (2) an awareness of apparent 
omissions and lack of next higher grade level reinforcement 
of important social studies concepts; and (3) a renewed 
realization on the part' of the local school administrators 
of the need for increased and improved supervision and 
monitoring of classroom instruction as a component of the 
district's quality control measures. 

The thira application is currently underway with the 
1976-77 NAEP science released items. This approach is 
similar to the aforementioned social studies plan. The 
age 9 and 13 items, along with the additional 
distric t-develO/ped items, were randomly assigned to four 
test sets and administered to students at the end of 
grades 4, 5, 8 and 9» The findings are to be reviewed by a 
committee of elementary and junior high teachers. This is 

to. be followed' by a review of the district's prescriptive 

.'J 

and descriptive science curricula. 

In the way, of summary the experiences in this district 
with the NAEP materials can be characterized as an aid in • 
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the effort to improve ln'StTi3'c"tttmiarl--pT-"agr^ 

In several ways. First, the NAEP objectives and released 



test items provided a direct connection to a nationwide base 
for identifying, extending and/or confirming local 
instructional program objeccives. This has served to, 
partially negate some of the in sular . thinking that seems to 
be indigenous to small-district program planning and 
development. More often than not, experience has shown over 
the past years an acceptance on , the part of staff memb.ers of 
outside, prestigious sources of information . 

.'Second, the NAEP-developed objectives in the various 

subject areas are well-considered both in terms of scope and 

level so as to be useful as program goal indicators. They 

can thereby serve as an aid in maintaining a focus on the 
\. . ....... 

inttent of the programs. In a somewhat zealous need to 

' \ ■ ' * ■ . - 

develop learning or outcome objectives, there is a tendency 

to focu's^ on easy-to-measure trivial and lower-level 

cognitive\learnings at the expense of higher-level and 

\ 

affective o\itcomes. In this respect the goals or real 

intent of the\ programs of ten can be lost. 

" \ ■ 
\ 

\ 

Third, there lias been a cost-effective contribution in 
utilizing ^fAEP material. While there are still adamant ^ 
defenders of the loca^l curriculum turf and needs to reinvent 



\ 
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materials, the quari't'y of these' efforts , while notable, is 
at odds with the availability of the district's resources. 
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USING NAT]fONAL. ASSESSMENT AS A RESOURCE" 



-peR-E-VA-bUAmiG-AN D UP&A^>I-NG~&Ufi-R-IG-ULrUM- 



IN^RUCTIONAL MATERIALS AND TEACHING 

^ David R. Zitlow 

Social Studies Curriculum Coordinator 
Whitefish Bay High School 
1200 East Fairmont Avenue 
Whitefish Bay, Wisconsin 53217 



Whitefish Bay High School 'is a four-year public high 
school, located in an affluent suburb of Milwaukee, 
Wisconsin. Approximately 90 percent of the graduates 
continue .their^ edu'cati'on after graduatipn at four-year 
colleges and universities. The curriculum emphasis is , 
clearly college preparatory. 



On a periodic b^asis — usually five-year intervals * 
specific curriculum areas are scheduled for a systematic 
study and assessment for the purposes of evaluating and 
updating curriculum; instructional materials and teaching. 
During the 1980-P1 School year, the social studies program 
(K-12) was the focus of such a study. Areas of strengths 
and weaknesses were identified and concerns expressed. 
Recommendations for improvement were made, in the context of 
.the district's educational philosophy, emerging trends in 
social studies education, sound principles of learning and 
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concern foF^h^^ to funcrtl"b'n- 



adequately in the/|21s\t century. 



-it 



\ 



Early in the |tudy, )?ocial studies staff expressed a 



^serious concern /lover the\adequacy of ♦curriculum content , 

• • ' ■ I \ 

mater ialsr'and iristr*uction in prov id ing " stud ents with mastery 

- . J ■ \ ■ ' • 

of basic social, studies skills. In response to this 

concern, the social studies staff proposed a three-pronged 

approach: j- | 

1. formulating a d istinct statement of essential learner 
outcomes i?^ soc ial stud ies skills that the curr iculum 

should emffhasize; ^ 

/ • ^ - • ' 

2. d6'sig/ning. a skills-assessment instrument to be 

• administered to graduating 12th^grade students to 
determine student masteryof tl>e skills and processes 
objectives; and 

i ' ' 

3. modifying curriculum • and instruction to address 
inadequacies/revealed by the 'results of the skills 
assessment / . o ' 



The sobial studies curriculum study committee 
identified eight major categories of^^ssential social 
studies skills together with several subskills within 
each of these major categories — that students should 
have mastered upon graduating" fr.om Whitefish Bay High 



School. The list of these social studies skills 
objectives is shown in the Appendix. 

Next, the' author devised ap instrument comprising 
specific tast items assessing each subskill within each 
major skill category. At this point,' it was found that 
test items and statistical data released following ^ 
assessments conducted by National Assessment of ^ 
Educational Progress (NAEP) were extremely useful as a 
source for well-designed, credible questions. NAEP 
produces packets of>released test items with accompanying 
descriptive material |for each item: objective and 
subobjective , whether the item was group or individually 
administered, amount of administration time allocated to 
each item, age group(s) to which the item was 
administered, how the item was scored and percent correct 
for each item by- various reporting groups. NAEP items 
from the 1971-72 and the 1975-76 citizenship and social 
studies assessments were reviewed to ascertain whether 
they corresponded to the skills objectives identified by 
the local social studies curriculum study group. Of the 
75 items on the local skills-assessment instrument, about 
one-third were from NAEP, 

/' 

/ 

- / 




The instrument was administered to all graduating 
seniors in late May 1981. The student responses (class 
results and student response matrix ) were computer scored 
and a. question-by-question analysis was performed by the 
author. For the NAEP test items in the assessment 
instrument, it/ was possible to. compare the local group of 
examinees with a national sample of examinees. The 
baseline data provided with the NAEP test items enabled us 
to make some/meaningful comparisons of Whitefish Bay 
students with a correspond ing national sample . 



\ 



The item analysis, reveal ed some major deficiencies in 
projected student mastery of basic social studies skills. 
Lack of mastery was most evident in skills related to 
distinguishing relevant from irrelevant sources Of 
information, appraising judgments and values that are 
inv^lvejj in choosing a course ^of action and making 
inferences from data. However, for the most part, the 
students demonstra^ted a satisfactory level of mastery of 
social studies Skills. During the 1981-82 school year, 
staff W va/rious grade levels in social stud ies ^liXve been 
working together -t-o' design and select curriculum materials 

I i 

and. learning experiences that will compensate for the 

inadequavcies pointed out by the assessment. These new 

r /> . 

approaches and materials will be implemented during the 
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next school year. In addition, it is planned to 
^ readminister the skills-assessment instrument in the 
I future to determine if student proficiency at the mastery 
^level has been achieved in the areas previously identified 

as deficient. 

Based on my experiences with conducting a local 
assessment, I. would offer the following suggestions to 
those responsible for developing a locally developed 
objectives- referenced instrument: 

!• It is important that the objectives or competencies to 
be measured are well defined • I would recommend NAEP 
objectives — which benefit from the contributions by 
scholars in the f i'&ld , teachers and lay persons — as 
offering a useful framework for . identi f ication of local 
objectives. 

. 2. Released .items from theXNAEP assessments, together 
with available descriptive an^technical information, 
should be obtained to determineXif they can be used to 
measure any of the locally generat'e^ objectives to be 
assessed. Including NAEP items on a^ocal assessment 
instrument provides an invaluable way to compare the 
performance of local examinees with a nationalx sample • 
Other sources should be tapped for additional items to 
measure objectives for which NAEP items may not be 
suitable. 
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3. Local assessment committees should explore the 
possibility and/or desirability of getting consultant help 
in designing a local assessment program. Consultant 
services are available from NAEP, state departments of 
public instruction and local colleges and universities. 
^. A well-designed assessment instrument can serve as an 
excellent vehicle to diagnose and analyze weakness in a 
curricular program and can result in significantly 
improved programs and instruction. Instructional 
sequences can be targeted directly at the skills that have 
been tested to remediate student deficiencies. 
5. In the context of the current accountability movement, 
the undertaking of a local assessment opens the 
opportunity for local social studies staff to define and 
describe social studies outcomes in an unambiguous fashion 
through the use of test specifications. Modifications in 
instructional plans b'ased on the diagnosis of student 
strengths and weaknesses can serve as the basis of 
sustained and continuous improvement in the program of 
studies. 



The reader is invited to contact the author directly for 
copies of the skills-assessment instrument; results referred 
to in this article; and information related to designing, . 
administering and analyzing a local social studies ' 
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assessment. The following references are highly recommended 
for helpful information on assessing local social studies 
programs: 

Citizenship and Social Studies Objectives . Denver, Colo.: 
National Assessment of Educational Progress, Education 
Commission of the States, 1980. 

Citizenship Objectives for 197^-75 Assessment . Denver, \ 
Colo.: National Assessment of Educational Progress, ! 
Education Commission of the States, 1972. 

1 

Fair, J. (Ed.). National Assessment and Social Studies \ 
Education: A Review of Assessments in Citizenship and Socia) 
Studies by the National Council for the Social Studies . 
Washington, D.C.: U. S. Government Printing Office, 1975<. 

Fox, K.F., Williams, P. L. and Winters, L. "Graduation 
Competency Testing in the Social Studies: A Position , 
Statement of the National Council for the Social Studies." 
In Social Education , Vol. 43, May, 1979, PP. 367-372. 

Morse, H. T. and McCune, G. H. Selected Items for the 
Testing of Study Skills and Critical Thinking . Bulletin 15. 
Washington, D. C. : National Council for the Social Studies, 
1964. 
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Popham, W, J. Cr It er Ion « Referenced Measurement , Englewood 
Cliffs, N. J.: Prentice-Hall^ 1978. 

Williams, Paul L. and Moore, J. R. (Eds.). 
Criterion-Referenced Testing for the Soc ial St ud ies , 
Bulletin 6^. Washington, D. C. : National Council for the 
Social Studies, 1980. 
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APPENDIX SOCIAL STUDIES SKILLS OBJECTIVES 



A. Identifies the central problem in a situation; 
identifies the major issue in a dispute. 

Clarifies vague and ambiguous terminology. 
2. Distinguishes among definitional, value and 
factual issues in a dispute. 

B. Applies divergent thinking in formulating hypotheses 
and generalizations capable of being tested. 

C. Identifies and locates sources of information and. 
evaluates the reliability and relevance of these 
sources . 

I 1. Identifies and locates sources of information 

appropriate to the task. 

2. Distinguishes between relevant and irrelevant 

sources. ^ . 

3. Distinguishes between reliable and unreliable ' 

sources. 

D. . Demonstrates ability to use reliable sources of 

information . 

1. Uses more than one source to obtain information., 

2. Identifies points of agreement and disagreement 
among the sources . 

3. Evaluates the quality of the available 
iniformation . 

E. Organizes, analyzes, interprets and synthesizes 
information obtained from various sources. 

'l. Identifies central elements in information. 

2. Classifies information. 

3. Distinguishes statements of fact from statements 
of opinion . 

4. Distinguishes statements of inference from 
statements of fact. 

5. Identifies stated opinions, biases and value 
judgments . 

6. Differentiates between points of view. 

7. Recognizes logical errors. 

8. Recognizes inadequacies or omissions in 
information. 
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9. Makes inferences from data. 

10. Identifies cause and effect relationships. 

1-1. Recognizes interrelationships among concepts. 

12. Identifies nature of sample. 

13. Identifies stated and unstated assumptions. 

14. Summarizes information. 

F. Uses summarized information to test hypotheses, draw 
conclusions, o f fer sol ut ions to problems , clar i f y 
issues or make predictions. 

G. Validates outcomes in inM^estiffBs^ion . 

1. Tests solutions to problems or issues when 
possible . 

2. Modifies solutions in light of new factors or 
considerations. 

3. Analyzes trends and modifies predictions when 
necessary. 

H. ' Appraiaes judgments and values that are involved in 

the choice of a course of action. 

1. Identifies and weighs conflicting values which 
serve as a contradicting criteria for judging 
courses of action. 

2. Develops a set of criteria for judging proposed 
courses of action in terms of actual ^ and 
projected consequences. , ^ \ 

3. Applies the established, criteria to a'ctual and 
projected consequences of a proposed course of 
action . 

4. Selects and defends a position or course of 
action consistent with the established criteria. 
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